{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p5b0PzpW1xJq"
      },
      "source": [
        "![nebullvm nebuly AI accelerate inference optimize DeepLearning](https://user-images.githubusercontent.com/38586138/201391643-a80407e5-2c28-409c-90c9-327795cd27e8.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-KdJPm7M05Jc"
      },
      "source": [
        "# Accelerate Tensorflow ResNet50 with Speedster"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T9xuwZEHzN2K"
      },
      "source": [
        "Hi and welcome 👋\n",
        "\n",
        "In this notebook we will discover how in just a few steps you can speed up the response time of deep learning model inference using the Speedster app from the open-source library `nebullvm`.\n",
        "\n",
        "We will\n",
        "1. Install Speedster and the deep learning compilers used by the library.\n",
        "2. Speed up a PyTorch ResNet50 without any loss of accuracy.\n",
        "3. Achieve faster acceleration on the same model by applying more aggressive optimization techniques (e.g. pruning, quantization) under the constraint of sacrificing up to 2% accuracy.\n",
        "\n",
        "Let's jump to the code."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KIeIvBPVLQuq"
      },
      "outputs": [],
      "source": [
        "%env CUDA_VISIBLE_DEVICES=0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HbFy2Aykz2Qo"
      },
      "source": [
        "### Installation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "48aljCHu14-H"
      },
      "source": [
        "Install Speedster:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QFQh3BVr1-GO"
      },
      "outputs": [],
      "source": [
        "!pip install speedster"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8a7a86b3"
      },
      "source": [
        "Install deep learning compilers:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cffbfa32"
      },
      "outputs": [],
      "source": [
        "!python -m nebullvm.installers.auto_installer --frameworks tensorflow --compilers all"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N5RXHoZl0p3p"
      },
      "source": [
        "## Optimization example with Tensorflow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Ju-VcRH01Mw"
      },
      "source": [
        "In the following example we will try to optimize a standard resnet50 loaded directly from keras.\n",
        "\n",
        "Speedster can accelerate neural networks without loss of a user-defined precision metric, e.g. accuracy, or can achieve faster acceleration by applying more aggressive optimization techniques, such as pruning and quantization, that may have a negative impact on the selectic metric. The maximum threshold value for accuracy loss is determined by the metric_drop_ths parameter. Read more in the [docs](https://docs.nebuly.com/modules/speedster/getting-started).\n",
        "\n",
        "Let first test the optimization without accuracy loss (metric_drop_ths=0, default value), and then apply further accelerate it under the constrained of losing up to 2% of accuracy (metric = \"accuracy\", metric_drop_ths = 0.02)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "skxEuemn171G"
      },
      "source": [
        "### Scenario 1 - No accuracy drop"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wVRLXrDi2VaG"
      },
      "source": [
        "First we load the model and optimize it using the Speedster API:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2RbgGruAeQcf"
      },
      "outputs": [],
      "source": [
        "# If you encountered any error, run the cell again\n",
        "import tensorflow as tf\n",
        "from tensorflow.keras.applications.resnet50 import ResNet50\n",
        "from speedster import optimize_model, save_model, load_model\n",
        "\n",
        "# Load a resnet as example\n",
        "model = ResNet50()\n",
        "\n",
        "# Provide an input data for the model    \n",
        "input_data = [((tf.random.normal([1, 224, 224, 3]),), tf.constant([0]))]\n",
        "\n",
        "# Run Speedster optimization\n",
        "optimized_model = optimize_model(\n",
        "  model, input_data=input_data, optimization_time=\"unconstrained\"\n",
        ")\n",
        "\n",
        "# Try the optimized model\n",
        "x = tf.random.normal([1, 224, 224, 3])\n",
        "res_original = model.predict(x)\n",
        "res_optimized = optimized_model.predict(x)[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NGrk6_jwRubP"
      },
      "source": [
        "We can print the type of the optimized model to see which compiler was faster:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cVMn6erJLQuu"
      },
      "outputs": [],
      "source": [
        "optimized_model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aT0BhdIKR7gY"
      },
      "source": [
        "In our case, the optimized model type was TensorflowNvidiaInferenceLearner, so this means that Tensor RT was the faster compiler."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JMiuufyu2gD3"
      },
      "source": [
        "After the optimization step, we can compare the optimized model with the baseline one in order to verify that the output is the same and to measure the speed improvement"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Swpr-Wi5Si9a"
      },
      "source": [
        "First of all, let's print the results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MjGtKkeZSOc7"
      },
      "outputs": [],
      "source": [
        "res_original"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dhe94Tk3SSfn"
      },
      "outputs": [],
      "source": [
        "res_optimized"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UqxzStjD2v0r"
      },
      "source": [
        "Then, let's compute the average latency of the baseline model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ELyTjg6_S4Us"
      },
      "outputs": [],
      "source": [
        "import time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dkt67_Orwlv4"
      },
      "outputs": [],
      "source": [
        "num_iters = 100\n",
        "\n",
        "# Warmup\n",
        "for i in range(10):\n",
        "  model.predict(x)\n",
        "\n",
        "start = time.time()\n",
        "for i in range(num_iters):\n",
        "  model.predict(x)\n",
        "stop = time.time()\n",
        "\n",
        "print(\"Average latency original model: {:.4f} seconds\".format((stop - start) / num_iters))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AgOv-GqQ3KIC"
      },
      "source": [
        "Finally we compute the average latency for the optimized model:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4PodpaDVfwzT"
      },
      "outputs": [],
      "source": [
        "# Warmup\n",
        "for i in range(10):\n",
        "  optimized_model.predict(x)\n",
        "\n",
        "start = time.time()\n",
        "for i in range(num_iters):\n",
        "  optimized_model.predict(x)\n",
        "stop = time.time()\n",
        "\n",
        "print(\"Average latency optimized model: {:.4f} seconds\".format((stop - start) / num_iters))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tBeRKNTI3iyK"
      },
      "source": [
        "### Scenario 2 - Accuracy drop"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w3wutIzfAMe_"
      },
      "source": [
        "In this scenario, we set a max threshold for the accuracy drop to 2%"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fO1nGqpj3p7z"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.applications.resnet50 import ResNet50\n",
        "from speedster import optimize_model\n",
        "\n",
        "# Load a resnet as example\n",
        "model = ResNet50()\n",
        "\n",
        "# Provide an input data for the model   \n",
        "# Note that in this case we should provide the model at least 100 data samples\n",
        "input_data = [((tf.random.normal([1, 224, 224, 3]),), tf.constant([0])) for i in range(100)]\n",
        "\n",
        "# Run Speedster optimization\n",
        "optimized_model = optimize_model(\n",
        "  model, input_data=input_data, optimization_time=\"unconstrained\", metric = \"accuracy\", metric_drop_ths = 0.02\n",
        ")\n",
        "\n",
        "# Try the optimized model\n",
        "x = tf.random.normal([1, 224, 224, 3])\n",
        "res_original = model.predict(x)\n",
        "res_optimized = optimized_model.predict(x)[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yfW9kmHX-pGi"
      },
      "source": [
        "Here we compute the average throughput for the baseline model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0MMrL3959hli"
      },
      "outputs": [],
      "source": [
        "num_iters = 100\n",
        "\n",
        "# Warmup\n",
        "for i in range(10):\n",
        "  model.predict(x)\n",
        "\n",
        "start = time.time()\n",
        "for i in range(num_iters):\n",
        "  model.predict(x)\n",
        "stop = time.time()\n",
        "\n",
        "print(\"Average latency original model: {:.4f} seconds\".format((stop - start) / num_iters))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i3GqasOM-u8f"
      },
      "source": [
        "Here we compute the average throughput for the optimized model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_IbAW0KA4Fm5"
      },
      "outputs": [],
      "source": [
        "# Warmup\n",
        "for i in range(10):\n",
        "  optimized_model.predict(x)\n",
        "\n",
        "start = time.time()\n",
        "for i in range(num_iters):\n",
        "  optimized_model.predict(x)\n",
        "stop = time.time()\n",
        "\n",
        "print(\"Average latency optimized model: {:.4f} seconds\".format((stop - start) / num_iters))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4XFMC1S6zXTU"
      },
      "source": [
        "## Save and reload the optimized model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OXHVr3EAzbT5"
      },
      "source": [
        "We can easily save to disk the optimized model with the following line:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3M565P-zzaFB"
      },
      "outputs": [],
      "source": [
        "save_model(optimized_model, \"model_save_path\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ee8CS_Evzg1j"
      },
      "source": [
        "We can then load again the model:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zOQ88SY_zg-A"
      },
      "outputs": [],
      "source": [
        "optimized_model = load_model(\"model_save_path\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "id": "b77ff2ac",
      "metadata": {
        "id": "b77ff2ac"
      },
      "source": [
        "<center> \n",
        "    <a href=\"https://discord.com/invite/RbeQMu886J\" target=\"_blank\" style=\"text-decoration: none;\"> Join the community </a> |\n",
        "    <a href=\"https://nebuly.gitbook.io/nebuly/welcome/questions-and-contributions\" target=\"_blank\" style=\"text-decoration: none;\"> Contribute to the library </a>\n",
        "</center>\n",
        "\n",
        "<center> \n",
        "    <a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/speedster#key-concepts\" target=\"_blank\" style=\"text-decoration: none;\"> How speedster works </a> •\n",
        "    <a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/speedster#documentation\" target=\"_blank\" style=\"text-decoration: none;\"> Documentation </a> •\n",
        "    <a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/speedster#quick-start\" target=\"_blank\" style=\"text-decoration: none;\"> Quick start </a> \n",
        "</center>"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3.8.10 64-bit",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.10"
    },
    "vscode": {
      "interpreter": {
        "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}
